The present invention relates to an image processing apparatus for processing a color image signal and an image capturing apparatus including the functionality of the image processing apparatus and, in particular, to an image processing apparatus and an image capturing apparatus including a signal processing circuit having the functionality to process a plurality of image signals of predetermined filter components in parallel.
Image capturing apparatuses, such as digital video cameras and digital still cameras, include image sensors (e.g., charge coupled devices (CCDs) or complementary metal oxide semiconductor image sensors (CMOS sensors)). In general, these image sensors read out signals from two-dimensionally arranged pixels, converts the signals into one data stream, and outputs the data stream from an output channel. For example, in a known CCD, image signals are delivered to a vertical register provided to each of a plurality of columns. Subsequently, these signals are delivered to a horizontal register on a row-by-row basis and are output. Thus, these signals are converted to one data stream. In contrast, in CMOS sensors, for example, image signals for one of a plurality of rows are read out and stored in capacitors provided to all of columns. The signals are sequentially output from the first capacitor or the last capacitor so as to be converted to one data stream.
FIG. 33 is a block diagram of an exemplary structure of a main portion of an image capturing apparatus including a 1-channel output image sensor.
As shown in FIG. 33, an analog front-end (AFE) circuit 912 includes a correlated double sampling (CDS) circuit, an auto gain control (AGC) circuit, and an analog-to-digital (A/D) conversion circuit. The analog front-end (AFE) circuit 912 converts an analog image signal from an image sensor 911 to a digital signal and outputs the digital signal. A camera signal processing circuit 913 performs a variety of camera signal processing (e.g., digital clamp, noise reduction, defect correction, demosaic (synchronization), white balance correction, and resolution conversion) on the image signal from the AFE circuit 912. Finally, the camera signal processing circuit 913 outputs the image signal to a baseband processing circuit 914 in the form of a luminance signal (Y) and a color-difference signal (C) . The baseband processing circuit 914 performs processing relating to a baseband (e.g., compression coding of the input image signal and conversion of the signal to a monitor display signal).
A color sequence of an image signal input to the camera signal processing circuit 913 is discussed below with reference to a Bayer array format, which is widely used as filter coding of an image sensor. FIGS. 34A and 34B are diagrams illustrating a color sequence when pixel signals of an image sensor having the Bayer array format are output from one output channel.
As shown in FIG. 34A, in an image sensor having the Bayer array format, pixels are arranged so that R and Gr are alternately disposed from the head of odd rows and Gb and B are alternately disposed from the head of even rows. When only one output channel is provided, pixels are sequentially scanned in the horizontal direction from the upper left of this pixel array. When a 1H period is completed, the next row is scanned in the same manner so that image signals are read out. Accordingly, as shown in FIG. 34B, the color sequence of a pixel signal output from the image sensor 911 to an output channel Ch1 is “R, Gr, R, Gr . . . ” for an odd H period and “Gb, B, Gb, B . . . ” for an even H period. Therefore, the camera signal processing circuit 913 that performs processing for individual colors needs to recognize such a color sequence and appropriately carry out various processes for an R color, G color, and B color of the input signal in synchronization with the color sequence.
In recent years, image capturing apparatuses including an image sensor with a large number of pixels exceeding a million pixels have been increasingly used. In such an image sensor, since the number of pixels is large, a readout frequency of pixels becomes significantly high for the above-described 1-channel readout method. In particular, when, like a video camera, the readout time for one screen is limited, a high frequency is required. For example, if the readout frequency becomes too high, the consumption power is increased. In addition, due to the limitation of analog signal processing, the reading out of the signal may be impossible.
To solve such a problem, in recent years, camera signal processing systems capable of reading out pixel signals of an image sensor via multiple channels have been developed. If the camera processing systems can read out pixel signals from the image sensor via multiple channels, the readout frequency for one channel can be reduced. Thus, the above-described problem can be solved.
If the multichannel readout is employed, a multichannel-specific problem (such as uneven levels of analog signals from the channels) may arise. Accordingly, a technique that detects a difference of signal levels between channels and corrects the difference (refer to, for example, Japanese Unexamined Patent Application Publication No. 7-75019, in particular, paragraphs [0013] to [0016] and FIG. 1) and a technique that detects that difference on the basis of the integral value with respect to a plurality of rows (refer to, for example, Japanese Unexamined Patent Application Publication No. 2002-252808, in particular, paragraphs [0020] to [0033] and FIG. 1) have been proposed.
Examples of the color sequence when multiple output channels are used in an image sensor of the Bayer array format are given below. FIGS. 35A-B through 37A-B are diagrams illustrating color sequences when pixel signals are read out via 2 channels, 3 channels, and 4 channels, respectively.
In FIG. 35A, pixel signals starting from the first pixel are alternately output to two output channels Ch1 and Ch2. In this case, as shown in FIG. 35B, by simultaneously reading out two pixels (one for the channel Ch1 and one for the channel Ch2) in one clock cycle, the readout frequency can be reduced by ½. Additionally, in FIG. 36A, pixel signals starting from the first pixel are sequentially assigned to three output channels Ch1, Ch2, and Ch3 and are output to these output channels. In this case, as shown in FIG. 36B, by simultaneously outputting signals from three pixels in one clock cycle, the readout frequency can be reduced to ⅓. In FIG. 37A, pixel signals starting from the first pixel are sequentially assigned to four output channels Ch1 to Ch4 and are output to the four output channels. In this case, as shown in FIG. 37B, by simultaneously outputting signals from four pixels in one clock cycle, the readout frequency can be reduced to ¼.
Furthermore, as shown in FIG. 38A through 40B, a technique for simultaneously reading out pixel signals from a plurality of rows to the corresponding output channels has been proposed. FIGS. 38A-B through 40A-B illustrate color sequences in the cases where pixel signals for 2 rows are read out via 2, 4, and 6 channels, respectively.
As shown in FIG. 38A, the output channel Ch1 is assigned to odd rows of the image sensor whereas the output channel Ch2 is assigned to even rows of the image sensor. In this case, as shown in FIG. 38B, by simultaneously reading out pixel signals from 2 rows in one clock cycle, the readout frequency can be reduced by ½.
In FIG. 39A, the output channels Ch1 to Ch4 are assigned to sets of 2 rows×2 columns of the image sensor. In this example, signals from rows 1 and 2 of a column 1 are output to the channels Ch1 and Ch2, respectively. In addition, signals from rows 1 and 2 of a column 2 are output to the channels Ch3 and Ch4, respectively. In this case, as shown in FIG. 39B, by simultaneously reading out 4 pixel signals in one clock cycle, the readout frequency can be reduced to ¼.
In FIG. 40A, the output channels Ch1 to Ch4 are assigned to sets of 2 rows×3 columns of the image sensor. In this example, signals from rows 1 and 2 of a column 1 are output to the channels Ch1 and Ch2, respectively. In addition, signals from rows 1 and 2 of a column 2 are output to the channels Ch3 and Ch4, respectively. Signals from rows 1 and 2 of a column 3 are output to the channels Ch5 and Ch6, respectively. In this case, as shown in FIG. 40B, by simultaneously reading out 6 pixel signals in one clock cycle, the readout frequency can be reduced to ⅙.
It is noted that, for image sensors of an XY address scanning type (e.g., the CMOS sensors), the above-described readout operations can be easily achieved without significantly changing the basic structure thereof.
As noted above, by providing a plurality of output channels, the total number of clock cycles required for reading out pixel signals for one screen can be reduced, and therefore, the readout frequency can be reduced. However, as noted above, a plurality of output sequences are possible depending on the number of channels and depending on which pixel groups are read out in parallel. An appropriate signal processing circuit needs to be developed for the selected output sequence and needs to be mounted in the camera.
For example, as shown in FIGS. 39A and 39B, four filter components R, Gb, Gr, and B repeatedly appear during a readout operation. If this repeat number “4” (or an integral multiple of “4”) is equal to the number of output channels, one output channel can output a signal of the same filter component in any H period. Thus, the downstream circuits (e.g., the camera signal processing circuit 913 shown in FIG. 33) can process signals of the same filter component at a time. Accordingly, the circuit configuration can be simplified. However, in many cases (e.g., the case shown in FIGS. 40A and 40B), this condition is not satisfied. Due to the restrictions relating to the number of output channels and the readout frequency based on the circuit scale and the manufacturing cost, it is not easy to design the circuit that always satisfies the above-described condition.
Additionally, a system has been proposed in which multichannel signals output from an image sensor are not directly delivered to the downstream circuit. The system multiplexes the signals to reduce the number of channels and delivers the signals. FIG. 41 is an exemplary block diagram of a signal processing system having such multiplexing functionality.
Unlike the configuration shown in FIG. 33, the configuration of the signal processing system shown in FIG. 40 has a multiplexer (MUX) function in an AFE circuit 912a. The AFE circuit 912a multiplexes signals output from N channels into signals for, for example, N/2 channels (N: an integer greater than 2). For example, by time-multiplexing signals from two adjacent channels, the AFE circuit 912a can reduce the number of output channels. Although such a multiplexing operation doubles the output frequency, this high frequency can be achieved by multiplexing the signals after the signals are converted to digital signals.
In this multiplexing operation, since a plurality of combinations of the multiplexed channels appear, a signal processing circuit corresponding to each of a plurality of output sequences needs to be developed in addition to supporting a large number of output channels and a large number of parallel readout patterns from the above-described image sensor.
As described above, there are many patterns of the output sequence of pixel signals from the image sensor. When the outputs are multiplexed, the number of the patterns is further increased. All of the blocks of the downstream signal processing circuit need to operate so as to support all the output sequences. To support all the above-described output sequences, the configuration of the signal processing circuit becomes large-scaled and significantly complicated.
In recent years, in order to reduce the development and manufacturing costs when introducing new types of image capturing apparatuses to market, there has been a growing demand for design of a signal processing circuit that can support different types of image sensors (e.g., image sensors of different pixel resolutions) for several years without requiring changing of the basic configuration thereof. However, it is difficult to determine all of pixel-signal readout methods that are anticipated to be used for the next few years. Accordingly, the supporting specifications are limited.